Nonasymptotic quasi-optimality of AIC and the slope heuristics in maximum likelihood estimation of density using histogram models

We consider nonparametric maximum likelihood estimation of density using linear histogram models. More precisely, we investigate optimality of model selection procedures via penalization, when the number of models is polynomial in the number of data. It turns out that the Slope Heuristics …rst formulated by Birgé and Massart [10] is satis…ed under rather mild conditions on the density to be estimated and the structure of the considered partitions. This suggests a new look at AIC penalty and more precisely, we show that the minimal penalty in the sense of Birgé and Massart is equivalent to half AIC penalty. Thus, as soon as the chosen penalty is larger than half AIC, the model selection procedure satis…es an oracle inequality. On contrary, if the penalty is less than the minimal one, then the procedure totally misbehaves. Moreover, if the penalty is equal to AIC penalty and the number of data is large enough -, then the model selection procedure is nearly optimal in the sense that it satis…es a nonasymptotic, trajectorial oracle inequality with constant almost one, tending to one when the number of data goes to in…nity. Finally, it is, to our knowledge, the …rst time that the Slope Heuristics is theoritically validated in a non-quadratic setting. Keywords: Maximum likelihood, density estimation, AIC, Optimal model selection, Slope heuristics, Penalty calibration. 1 Introduction This paper is devoted to the study of some penalized maximum likelihood model selection procedures for the estimation of density on histograms. There is a huge amount of literature on the problem of model selection by penalized maximum likelihood criteria, even in the more restrictive question of selecting an histogram, that goes back to Akaike’s pioneer work. In the early seventies, Akaike [1] proposed to select a model by penalizing the empirical likelihood of maximum likelihood estimators by the number of parameters in each model. The analysis of Akaike [1] on the model selection procedure de…ned by the so-called Akaike’s Information Criterion (AIC), is fundamentally asymptotic in the sense that the author considers a given …nite collection of models with the number of data going to in…nity. This asymptotic setting is irrelevant in many situations and thus many e¤orts have been made to develop nonasymptotic analysis of model selection procedures, letting the dimension of the models and the cardinality of the collection of models depend on the number of data. As pointed out by Boucheron and Massart [11], it is nevertheless worth mentioning that early works of Akaike [2] and Mallows [22] in model selection relied, although in a disguised form, on the Wilks’ phenomenon (Wilks [28]) that asserts that in smooth parametric density estimation the di¤erence between the maximum likelihood and the likelihood of the sampling distribution converges towards a chi-square distribution where the number of degrees of freedom coincides with the model dimension. This phenomenon has been generalized by Boucheron and Massart [11] in a nonasymptotic way, considering the empirical excess risk in a M-estimation with bounded contrast setting, and is actually one the main results supporting the conjecture that the slope heuristics introduced by Birgé and Massart [10] hold in some general framework, see Arlot and Massart [6]. Let us now describe some works related to the selection of maximum likelihood estimators. Barron and Sheu [9] give some risks bounds on maximum likelihood estimation considering sequences of regular exponential families made of polynomials, splines and trigonometric series. They achieve an accurate